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0 Detection of global translations between images. 
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0 Described herein is a method to detect global displacements between pairs of images to sul>pixel accuracy 
using a Fourier domain correlation based on partial normalisation of the product of the magnitudes and the 
phase difference between the pairs of images. 
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Field of the Invention 

The present invention relates to the detection of global translations between Images and is more 
particularly concerned with a digital processing method for such detection to sub-pixel accuracy between 
5 images which are related either temporally or spatially. 

Baclcground of the invention 

Images may, for example, comprise part or all of a time-varying sequence, or the colour separations of 
10 a single image. Typical applications for the method of the present Invention include the automatic detection 
and correction, to high accuracy, of spatial unsteadiness occurring in a time-varying sequence of images 
which have originated from motion picture film, the automatic alignment of pairs of images of the same 
subject taken at different times, and the automatic registration of colour separations of a single image. 
There are two classes of algorithms by which spatial translation can be measured, the first relating to 
75 the use of non-image Information, for example, the position of sprocket holes or the frame edge or the use 
of fiducial marks, and the second depending on the use of the information content within the image. The 
second class of algorithms can be referred to as 'motion estimation' algorithms. The present Invention 
belongs to this second class. 

Motion estimation algorithms have the advantage of versatility in this particular application. They can be 
20 Implemented on any image set independent of its source or history. 

Motion estimation algorithms may be broadly divided into three types:- 

1. Block matching algorithms. 

25 These are described in "A Hardware Motion Compensator for a Videoconferencing Codec", I Parke 
and D G Morrison, and "Motion -compensated Field Rate Conversion for HDTV Display", G M X 
Fernando, both from IEEE Colloquium "Motion Compensated Image Processing", Digest 1987/86, and "7/)e 
Development of an HDTV to PAL Standards Converter", T Ohmura, IBC86 Conference Proceedings, 
pages 221 to 224. 

30 In this method, the matching is implemented in the spatial domain, based on either maximum cross- 
correlation or minimum mean square error. The method is amenable to efficient hardware implementation 
by the application of hierarchical searching techniques. 

2. Spatio-temporal constraint or gradient matching algorithms. 

35 

These are described in "Motion Estimation for Frame-rate Conversion", E A Krause, MIT PhD 
Thesis, ATRP-T-70 (1987). 

The method used here is based on the assumption that movement occurs along paths of constant 
luminance. Thus, if luminance, L, Is a function of spatial position. and time, f, it Is required that:- 

40 

~WO.}'(0.0 = 0 

45 

Expanding this, and neglecting higher order terms yields: 
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may be obtained by placing constraints on the smoothness of the image flow, both spatially and temporally. 
Typically, the spatial and temporal gradients of the luminance signal are approximated from the output from 
local spatial high-pass filtering and the inter frame difference of corresponding pixel values respectively. 
This is described in "Methods for Measuring Small Displacements of Television Images'*, C Cafforio et 
5 al. IEEE Transactions on Information Theory, IT-22, 5, 573-579 (1976). 

3. Phase correlation algorithms. 

These are described in "TV Picture Motion Measurement", G A Thomas, WO-A-87/05769 and "Cine 
10 Unsteadiness in Video Signal Correction", K L Minakovic, GB-A-2 187 913. 

Phase correlation Is based in the principle that x-y translations in the spatial domain correspond to 
phase shifts In the Fourier domain. The phase shift at each frequency may be calculated as follows:- 



|f(«,v).G(«,v)*| 

where F and G are the two-dimensional Fourier transforms of blocks f,g in corresponding positions from 
20 neighbouring frames in the input sequence, or from colour separations of the same image. The inverse 
transform of A^{u,v) generates a correlation surface which contains spikes (more precisely, sine functions 
which result from the effective window in the Fourier domain) at positions corresponding to an x-y 
displacement, and where the height of the spikes is related to the number of pixels within the block which 
have moved with that displacement. 
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Problem to be solved by the Invention 



Methods utilising block matching algorithms as discussed above are not suitable for reliable measure- 
ment of displacement or translation to sub-pixel accuracy. 
30 The method used in spatio-temporal constraint or gradient matching algorithms is robust in generating 
vectors of sub-pixel accuracy only where the displacement is small - for example, of the order of one pixel. 
The principles which underpin this method render it suitable for the generation of motion vectors on a local 
(pixel by pixel) rather than a global (block by block) basis. Hence motion estimation techniques k>ased on 
the assumptions of optical flow are not appropriate for the detection of global translations. 
35 Phase correlation can be demonstrated to be extremely robust in the presence of frame-independent 
noise (which is spread across the entire correlation surface), and independent movement within the scene 
(which generally manifests itself as low-amplitude independent spikes in the correlation surface which can 
be separated without adverse effect on the spike corresponding to a global displacement). 

In principle, therefore, phase correlation appears to be the most suitable approach to estimate 
40 displacements where a single global vector is required per frame. 

However, phase correlation yields a correlation surface which is sampled at the sampling rate of the 
input data, and hence, in its simplest form, generates vectors to pixel accuracy. It is possible to perform a 
two-dimensional interpolation over the correlation surface in an attempt either to reconstruct a continuous 
correlation surface and subsequently to estimate the position of the peak to sub-pixel accuracy, or in a 
45 single step, to estimate the corresponding position of a peak In an equivalent continuous surface. 

It can be shown, however, that errors exist in the reconstructed surface due to edge effects and spectral 



In the former case, blocks are selected from the pair of images at corresponding pixel positions. As a 
consequence of the global displacement between the image pair, corresponding blocks contain picture 

50 Information from slightly different areas of the image. 

In the latter case, the signals input to the phase correlation are sampled, hence the phase-difference 
signals in the Fourier domain are repeated centred around multiples of the sampling frequency. If these 
'repeat spectra' are filtered out using a window with a cut-off frequency at the input nyquist frequency, then 
the Inverse transform of the window is effectively convolved with the sampled correlation surface. 

55 Alternatively, if the phase-difference signal with 'repeat spectra', and hence the discontinuities which occur 
at the nyquist frequency, is inverse transformed, these discontinuities produce errors in the correlation 
surface. As a result of these errors in the reconstructed surface, displacement vectors are generated which 
are only accurate to around 0.5 of a pixel. 
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Several methods have been proposed to improve this accuracy. For example, using the phase 
correlation algorithm discussed abovOi the Fourier domain can be padded with 'O's before the inverse 
transform, to effect a sine function interpolation in the spatial domain. However, this results in objectionable 
ringing in the spatial domain which may be reduced by windowing techniques, but the overall computational 
5 complexity of the method is high. 

Summary of the Invention 

It is therefore an object of the present invention to provide a comparatively simple method to extend the 
70 phase correlation technique to generate displacement vectors with an accuracy significantly better than 0.5 
of a pixel. In this case, the extension of the phase correlation technique comprises a Fourier domain 
correlation method. The image may then be realigned according to the global displacement vector by 
means of spatial interpolation to sub-pixel accuracy using a method as described, for example, in US-A-5 
243 433. 

75 In accordance with one aspect of the present invention, there is provided a method of detecting global 
translations between two Images comprising the steps of:- 

a) generating at least one displacement vector using partially normalised phase correlation techniques 
indicative of the translation between the two images to pixel accuracy using the equation:- 



20 



AO(m,v) = ; 



|F(M.v),G(«,v)t 



25 where 0<n<1; 

b) storing the correlation surfaces obtained in step a); 

c) selecting a single vector which most accurately represents the global translation between the two 
images; and 

d) interpolating the stored correlation surfaces to provide detection to sub-pixel accuracy. 

30 Preferably, step b) includes applying a threshold to the at least one displacement vector to provide an 
indication of suitable vector(s). 

The expression for phase correlation as discussed above may be re-written as follows:- 



|F(M,v).G(u.v)*|" 

where n is a normalisation factor. 
40 When n = 1. the magnitude is fully normalised and A^((/,v) comprises unity magnitude and a phase 
difference at each frequency. This is the most significant step in the calculation of conventional phase 
correlation. 

When 77 = 0, there is no normalisation of the product of magnitudes and hence this expression 
corresponds to the most significant step in the calculation of cross-correlation implemented in the Fourier 
45 domain. 

In the present invention, a normalisation factor, n. which lies between the values of 0 and 1 is 
employed. Hence, the method is a weighted hybrid between phase correlation and Fourier domain cross- 
correlation as a function of n. 

In accordance with the present invention, partially normalised phase correlation is applied to a number 

50 of, corresponding sub-blocks of an appropriate size in the pair of images. A set of vectors is obtained from 
the generated correlation surfaces. This set may comprise all vectors which correspond to peaks of height 
greater than a pre-spectfied value, or the vectors which correspond to the peak of the greatest height for 
each correlation surface, or vectors selected according to some other appropriate measure. Of this set of 
vectors, a single vector is selected, by some appropriate measure, which most accurately represents the 

55 global displacement between the two frames being correlated to pixel accuracy. A two-dimensional 
quadratic interpolation is then performed on the correlation surface which generated that global displace- 
ment vector to achieve a vector to sub-pixel accuracy. 
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Advantageous Effect of the Invention 

The advantage of the method according to the present invention over those described in the prior art is 
that it takes advantage of the strengths particular to the phase correlation and cross-correlation techniques, 
5 and Is comparatively simple to Implement In hardware. 

As discussed above, phase correlation is extremely robust due to the spiky nature of the correlation 
' surface, to noise and independent movement within the scene. Hence vectors can be generated with high 
confidence - but only to pixel accuracy. 

Conversely, cross-correlation generates a correlation surface which is smooth and substantially less 
10 robust to noise. These techniques can hence be used to generate vectors with higher accuracy but low 
confidence. 

By using a weighted hybrid of phase correlation and cross-correlation techniques, errors in the the 
reconstructed correlation surface are reduced. Hence, vectors of sub-pixel accuracy may be obtained with a 
high degree of confidence. 



75 



Brief Description of the Drawings 



For a better understanding of the present invention, reference wilt now be made, by way of example 
only, to the accompanying drawings in which:- 
20 Figure 1 illustrates a block diagram of apparatus for carrying out the method of the present invention; 
and 

Figure 2 is a block diagram illustrating the various steps In the method according to the present 
invention. 

25 Detailed Description of the Invention 

As mentioned above, the present invention can be used to determine displacements or translations 
between pairs of image frames due to movement from frame to frame which is not due to movement of 

image detail. 

30 Furthermore, the translation to be measured may be from the colour separations of a single image and 
then the translation is required to effect the correct alignment of the colour images when the frames are 
reconstructed to produce a full colour final image. 

The details of a preferred implementation of the present invention for each pair of successive frames in 
sequence, or for a pair of separations from a colour change Is as follows:- 
35 1) Store blocks at corresponding positions in the paris of frames. This may be either five (corners plus 
centre) or nine (corners, centre edges plus centre) blocks each comprising 64 x 64 pixels. 

2) Perform a two-dimensional partially normalised phase correlation on each corresponding pair of blocks 
as given in equation (1) below:- 

40 

A<l>(u,v)= , — ^ (1) 
|F(«.v).G(ii,v)*r 

45 The optimum value for n which corresponds to the value which generates displacement vectors of 

the greatest accuracy have been found to lie in the range of 0.5 to 0.6. The inverse transform is then 
taken to obtain a correlation surface for each block pair. 

3) Threshold each correlation surface at a value corresponding to the minimum value expected for 
significant peaks (typically around 0.1 to 0.2). This will generally yield a significant vector at only one 

50 displacement position. Finally, use a search technique to find the position of the remaining peak (if any) 
with the greatest value. In blocks where there is very little Input signal or most of the block was in 
independent motion, there may bo no remaining peaks. 

4) Take the modal value from the resulting vector set (that is, the displacement vector which occurs most 
frequently). This Is the global motion vector (x.>) to pixel accuracy. 

55 5) For sub-pixel accuracy, quadratic interpolation is performed on the correlation surface which gen- 
erated the modal vector with maximum peak height (that is, with maximum confidence):- 
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10 



;c+5x=jc+ 



where 

Cb,o is the value of the correlation surface at the position of the global motion vector. 
Ci,o Is the value of one pixel to the right etc. 
The result is a single vector (x+ax.y+sy) for each pair of successive frames which corresponds to the 
75 sub-pixel displacement between them. 

Where displacement vectors are calculated for a sequence of images, calculations for the phase 
correlation may be reduced by storing the Fourier coefficients from the second frame of each pair for use 
when that frame becomes the first of the subsequent pair, or alternatively, by storing the Fourier coefficients 
for some reference frame in the sequence. 
20 Rgure 1 illustrates the overall apparatus which can be used to effect correction for global translation. 
The apparatus comprises an Image store 10, a phase correlation device 12, a reference frame stores 14 
and a spatial Interpolation unit 16. 

The reference frame store 14 store Fourier coefficients derived from a reference frame. This reference 
frame may be the first frame of a sequence of frames, the subsequent frames being connected to be In 
26 alignment with the first frame. 

The spatial interpolation unit 16 operates on the principle as described in US-A-5 243 433 discussed 
above, to provide the necessary correction. 

Once the Fourier coefficients values have been stored in the reference frame store 14, an input Image 
frame to which global translation correction is to be applied is Input into the image store 10. The input 
30 image Is then fed to the phase correlation device 12. Fourier coefficients values are input into the device 12 
from the store 14 so that correlation between the reference frame and the input frame can be achieved. 
After correlation in the phase correlation device 12, an output global displacement vector is obtained which 
gives the correction which Is to be applied to the input Image frame In the spatial Interpolation unit 16 to 
compensate for the translation between that Input image frame and the reference frame. 
35 The phase correlation device 12 may comprise a VME bus card and FFT/DSP chips programmed to 
carry out the desired correlation functions. 

Figure 2 illustrates the steps of the preferred method In more detail. Components of the apparatus 
already mentioned are referenced the same. 

At the start of any global translation correction, a reference frame is chosen against which all 
40 subsequent frames are aligned. The first frame, the reference frame, is input to the frame store 10, and is 
fed to the phase correlation device 12 Indicated by the dotted line In Figure 2. The first step carried out on 
this frame is the selection of sub-blocks values as shown by box (A). These values are passed to box (B) 
where the Fourier coefficients are determined. These coefficients are stored in reference frame store 14 (as 
shown by line 100). The apparatus is now ready to detect global translation and then provide the global 
45 displacement vector for feeding to the spatial interpolation unit 16. 

An input image frame to be connected Is Input to the frame store 10. As with the reference frame. It Is 
input to the phase correlation device 12 where the sub-blocks are selected in box (A). Fourier transforma- 
tion of these sub-blocks is carried out in box (B) and then partially normalised phase correlation of the sub- 
block pairs is carried out in box (C) using the stored Fourier coefficients from reference frame store 14. This 
50 correlation provides a correlation surface with peaks corresponding to correlation between the input image 
frame and the reference Image frame. A threshold is then applied to the correlation surface to determine the 
peaks, as described previously, In box (D). The modal vector is then determined in box (E) from the peaks 
determined In box (D) together with the address of the block which generated the modal vector - the modal 
vector x,y being fed to box (F). 
55 The correlation surfaces obtained by the partially normalised phase correlation on each of the sub-block 
pairs in box (C) are stored in box (Q). 

Information from box (E) relating to the address of the block which generated the modal vector is 
determined in box (H) and fed to box (G). 
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Interpolation of the 'best' correlation surface is then carried out in box (I) to provide the fractional 
corrections ax,6y in box (J). 

The output from box (J) Is then fed into box (F) to provide the displacement vector for the spatial 
interpolation unit 16. The input image frame from the frame store 10 is also fed to the spatial interpolation 
5 unit to effect correction on the input Image. 

Claims 

1. A method of detecting global translations between two images comprising the steps of:- 
10 a) generating at least one displacement vector using partially normalised phase correlation tech- 

niques indicative of the translation between the two images to pixel accuracy using the equation:- 

^ |f(U,V).C(M.V)t 



where 0<n<1: 

b) storing the correlation surfaces obtained in step a); 
20 c) selecting a single vector which most accurately represents the global translation between the two 

images; and 

d) interpolating the stored correlation surfaces to provide detection to sub-pixel accuracy. 

2. A metiiod according to claim 1, wherein step c) includes applying a threshold to the at least one 
25 displacement vector to provide an indication of suitable vector(s). 

3. A method according to claim 1 or 2. wherein step c) includes selecting the modal value from the at 
least one generated displacement vector. 

30 4. A method according to any one of the preceding claims, wherein 0.5<ak0.6. 
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